Query generation for a capture system

ABSTRACT

A document accessible over a network can be registered. A registered document, and the content contained therein, is not transmitted undetected over and off of the network. In one embodiment, the invention includes a manager agent to maintain signatures of registered documents and a match agent to detect the unauthorized transmission of the content of registered documents.

RELATED APPLICATIONS

This Application is a divisional (and claims the benefit of priorityunder 35 U.S.C. §120 and §121) of U.S. application Ser. No. 12/690,153,filed Jan. 20, 2010, now U.S. Pat. No. 8,005,863, entitled “QUERYGENERATION FOR A CAPTURE SYSTEM,” Inventor(s) Erik de la Iglesia, etal., which is a continuation of U.S. patent application Ser. No.11/439,488, filed May 22, 2006, now U.S. Pat. No. 7,689,614, entitled“QUERY GENERATION FOR A CAPTURE SYSTEM,” Inventor(s) Erik de la Iglesia,et al. The disclosures of the prior applications are considered part of(and are incorporated by reference in) the disclosure of thisapplication.

FIELD OF THE INVENTION

The present invention relates to computer networks, and in particular,to registering documents in a computer network.

BACKGROUND

Computer networks and systems have become indispensable tools for modernbusiness. Modern enterprises use such networks for communications andfor storage. The information and data stored on the network of abusiness enterprise is often a highly valuable asset. Modern enterprisesuse numerous tools to keep outsiders, intruders, and unauthorizedpersonnel from accessing valuable information stored on the network.These tools include firewalls, intrusion detection systems, and packetsniffer devices.

FIG. 1 illustrates a simple prior art configuration of a local areanetwork (LAN) 100 connected to the Internet 102. Connected to the LAN100 are various components, such as servers 104, clients 106, and switch108. Numerous other networking components and computing devices areconnectable to the LAN 100. The LAN 100 may be implemented using variouswireline or wireless technologies, such as Ethernet and the 802.11 theIEEE family of wireless communication standards. LAN 100 could beconnected to other LANs.

In this prior configuration, the LAN 100 is connected to the Internet102 via a router 110. This router 110 may be used to implement afirewall. Firewalls are widely used to try to provide users of the LAN100 with secure access to the Internet 102 as well as to provideseparation of a public Web server (for example, one of the servers 104)from an internal network (for example, LAN 100). Data leaving the LAN100 to the Internet 102 passes through the router 110. The router 110simply forwards packets as is from the LAN 100 to the Internet 102.

However, once an intruder has gained access to sensitive content insidea LAN such as LAN 100, there presently is no network device that canprevent the electronic transmission of the content from the network tooutside the network. Similarly, there is no network device that cananalyse the data leaving the network to monitor for policy violations,and make it possible to track down information leeks.

Prior data storage techniques placed files/data in open locations of adisk. The location was not dependent on the time of the storage request.The location could be dependent on the relative importance (importantfiles and/or files likely to be retrieved from the disk frequently areassigned to inner-areas of the disk) and/or what space is open. FIG. 2illustrates an exemplary prior art disk storing three files (A, B, andC) with each file occupying three blocks of space. These files werestored in the order of A, then B, and finally C. Each of these filescontains files A and B contain the same text (“example_(—)1”) and file Ccontains text different than A and B (“example_(—)2”). File A is storedin an inner-area. Files B and C are image files that are not usedfrequently. File C is stored in an intermediate location of the diskthat was open. As illustrated, File B is stored in area farther out thanA or B and is not stored in contiguous blocks. If these files werestored based on time, then A would be an innermost-area, followed by B,and then C.

When searching for a particular file or files that were stored in theprior art storage technique the entire disk and/or file system (such asa file allocation table or FAT) was searched to find the desired file orfiles. FIG. 3 illustrates an exemplary prior art search using lists. Asshown, each block (or each line of the file system) is search seriallyuntil the desired information is located. When the information is foundit is added to a list of positive matches.

If the search was to determine what files have the text “example_(—)1”and “example_(—)2” two different lists (list A and list B) would becreated after each serial search is performed. After the two lists arecreated, the cross product (A×B) is performed and the results areevaluated. This means that the number of evaluations that have to beperformed is the number of matches of in A multiplied by the number ofmatches in B. In other words, when looking for more than one piece ofdata, the search becomes an order n² operation.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings in which likereference numerals refer to similar elements and in which:

FIG. 1 is a block diagram illustrating a computer network connected tothe Internet;

FIG. 2 is a block diagram illustrating one configuration of a capturesystem according to one embodiment of the present invention;

FIG. 3 is a block diagram illustrating the capture system according toone embodiment of the present invention;

FIG. 4 is a block diagram illustrating an object assembly moduleaccording to one embodiment of the present invention;

FIG. 5 is a block diagram illustrating an object store module accordingto one embodiment of the present invention;

FIG. 6 is a block diagram illustrating a document registration systemaccording to one embodiment of the present invention;

FIG. 7 is a block diagram illustrating registration module according toone embodiment of the present invention; and

FIG. 8 illustrates an embodiment of the flow of the operation of aregistration module;

FIG. 9 is a flow diagram illustrating an embodiment of a flow togenerate signatures;

FIG. 10 is a flow diagram illustrating an embodiment of changing tokensinto document signatures;

FIG. 11 illustrates an embodiment of a registration engine thatgenerates signatures for documents;

FIG. 12 illustrates an exemplary embodiment of a system for thedetection of registered content is performed on a distributed basis;

FIG. 13 shows an embodiment of a computing system (e.g., a computer);

FIG. 14 illustrates an exemplary flow of querying captured objects;

FIG. 15 illustrates an exemplary embodiment of a capture system forquerying captured data;

FIGS. 16( a)-(c) illustrate an embodiment of an exemplary storageconfiguration for data received by a capture system;

FIG. 17 illustrates an embodiment of a method for performing a query ina capture system;

FIG. 18 illustrates pipeline staggering;

FIG. 19 illustrates the exemplary query being performed; and

FIG. 20 shows an embodiment of a computing system.

DETAILED DESCRIPTION

Although the present system will be discussed with reference to variousillustrated examples, these examples should not be read to limit thebroader spirit and scope of the present invention. Some portions of thedetailed description that follows are presented in terms of algorithmsand symbolic representations of operations on data within a computermemory. These algorithmic descriptions and representations are the meansused by those skilled in the computer science arts to most effectivelyconvey the substance of their work to others skilled in the art. Analgorithm is here, and generally, conceived to be a self-consistentsequence of steps leading to a desired result. The steps are thoserequiring physical manipulations of physical quantities. Usually, thoughnot necessarily, these quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared and otherwise manipulated.

It has proven convenient at times, principally for reasons of commonusage, to refer to these signals as bits, values, elements, symbols,characters, terms, numbers or the like. It should be borne in mind,however, that all of these and similar terms are to be associated withthe appropriate physical quantities and are merely convenient labelsapplied to these quantities. Unless specifically stated otherwise, itwill be appreciated that throughout the description of the presentinvention, use of terms such as “processing”, “computing”,“calculating”, “determining”, “displaying” or the like, refer to theaction and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system's registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

Exemplary Networks

As described earlier, the router 110 of the prior art simply routespackets to and from a network and the Internet. While the router may logthat a transaction has occurred (packets have been routed), it does notcapture, analyze, or store the content contained in the packets.

FIG. 4 illustrates an embodiment of a system utilizing a capture device.In FIG. 4, the router 410 is also connected to a capture system 400 inaddition to the Internet 402 and LAN 412. Generally, the router 410transmits the outgoing data stream to the Internet 402 and a copy ofthat stream to the capture system 400. The router 410 may also sendincoming data to the capture system 400 and LAN 412.

However, other configurations are possible. For example, the capturesystem 400 may be configured sequentially in front of or behind therouter 410. In systems where a router is not used, the capture system400 is located between the LAN 412 and the Internet 402. In other words,if a router is not used the capture system 400 forwards packets to theInternet. In one embodiment, the capture system 400 has a user interfaceaccessible from a LAN-attached device such as a client 406.

The capture system 400 intercepts data leaving a network such as LAN412. In an embodiment, the capture system also intercepts data beingcommunicated internal to a network such as LAN 412. The capture system400 reconstructs the documents leaving the network 100 and stores themin a searchable fashion. The capture system 400 is then usable to searchand sort through all documents that have left the network 100. There aremany reasons such documents may be of interest, including networksecurity reasons, intellectual property concerns, corporate governanceregulations, and other corporate policy concerns. Exemplary documentsinclude, but are not limited to, Microsoft Office documents, text files,images (such as JPEG, BMP, GIF, etc.), Portable Document Format (PDF)files, archive files (such as GZIP, ZIP, TAR, JAR, WAR, RAR, etc.),email messages, email attachments, audio files, video files, source codefiles, executable files, etc.

Capture System

FIG. 5 shows an embodiment of a capture system in greater detail. Acapture system (such as capture system 400 or 512) may also be referredto as a content analyzer, content or data analysis system, or othersimilar name. For simplicity, the capture system has been labeled ascapture system 500. However, the discussion regarding capture system 500is equally applicable to capture system 400. A network interface module500 receives (captures) data from a network or router. Exemplary networkinterface modules 500 include network interface cards (NICs) (forexample, Ethernet cards). More than one NIC may be present in thecapture system 512.

Captured data is passed to a packet capture module 502 from the networkinterface module 500. The packet capture module 502 extracts packetsfrom this data stream. Packet data is extracted from a packet byremoving the headers and checksums from the packet. The packet capturemodule 502 may extract packets from multiple sources to multipledestinations for the data stream. One such case is asymmetric routingwhere packets from source A to destination B travel along one path butresponses from destination B to source A travel along a different path.Each path may be a separate “source” for the packet capture module 502to obtain packets.

An object assembly module 504 reconstructs the objects being transmittedfrom the packets extracted by the packet capture module 502. When adocument is transmitted, such as in email attachment, it is broken downinto packets according to various data transfer protocols such asTransmission Control Protocol/Internet Protocol (TCP/IP), UDP, HTTP,etc. The object assembly module 504 is able to reconstruct the originalor reasonably equivalent document from the captured packets. Forexample, a PDF document would be broken down into packets before beingtransmitted from a network, these packets are reconfigurable to form theoriginal (or reasonable equivalent) PDF. A complete data stream isobtained by reconstruction of multiple packets. The process by which apacket is created is beyond the scope of this application.

FIG. 6 illustrates an embodiment of an object assembly module. Thisobject assembly module 606 includes a reassembler 600, protocoldemultiplexer (demux) 602, and a protocol classifier 604. Packetsentering the object assembly module 606 are provided to the reassembler600. The reassembler 600 groups (assembles) the packets into at leastone unique flow. An exemplary flow includes packets with identicalsource IP and destination IP addresses and/or identical TCP source anddestination ports. In other words, the reassembler 600 organizes apacket stream by sender and recipient.

The reassembler 600 begins a new flow upon the observation of a startingpacket. This starting packet is normally defined by the data transferprotocol being used. For TCP/IP, the starting packet is generallyreferred to as the “SYN” packet. The flow terminates upon observing afinishing packet (for example, a “Reset” or “FIN” packet in TCP/IP). Ifthe finishing packet is observed by the reassembler 600 within apre-determined time constraint, the flow terminates via a timeoutmechanism. A TCP flow contains an ordered sequence of packets that maybe assembled into a contiguous data stream by the reassemble 600. Thus,a flow is an ordered data stream of a single communication between asource and a destination.

The flow assembled by the reassembler 600 is provided to a protocoldemultiplexer (demux) 602. In an embodiment, the protocol demux 602sorts assembled flows using ports, such as TCP and/or UDP ports, byperforming a speculative classification of the flow contents based onthe association of well-known port numbers with specified protocols. Forexample, Web Hyper Text Transfer Protocol (HTTP) packets (such as, Webtraffic packets) are typically associated with TCP port 80, FileTransfer Protocol (FTP) packets with TCP port 20, Kerberosauthentication packets with TCP port 88, etc. Thus, the protocol demux402 separates the different protocols that exist in a flow.

A protocol classifier 604 may further sort the flows in addition to thesorting done by the protocol demux 602. The protocol classifier 604(operating either in parallel or in sequence to the protocol demux 602)applies signature filters to a flow to attempt to identify the protocolbased solely on the transported data. Furthermore, the protocolclassifier 604 may override the classification assigned by the protocoldemux 402. The protocol classifier 604 uses a protocol's signature(s)(such as, the characteristic data sequences of a defined protocol) toverify the speculative classification performed by the protocol demux602. For example, if an individual or program attempted to masquerade anillicit communication (such as file sharing) using an apparently benignport (for example, TCP port 80), the protocol classifier 604 would usethe HTTP protocol signature(s) to verify the speculative classificationperformed by protocol demux 602.

An object assembly module, such as object assembly modules 504 and 606outputs each flow, organized by protocol, which represent the underlyingobjects being transmitted. These objects are passed to the objectclassification module 506 (also referred to as the “content classifier”)for classification based on content. A classified flow may still containmultiple content objects depending on the protocol used. For example, asingle flow using HTTP may contain over 100 objects of any number ofcontent types. To deconstruct the flow, each object contained in theflow is individually extracted and decoded, if necessary, by the objectclassification module 506.

The object classification module 506 uses the inherent properties and/orsignatures of various documents to determine the content type of eachobject. For example, a Word document has a signature that is distinctfrom a PowerPoint document or an email. The object classification module506 extracts each object and sorts them according to content type. Thisclassification prevents the transfer of a document whose file extensionor other property has been altered. For example, a Word document mayhave its extension changed from .doc to .dock but the properties and/orsignatures of that Word document remain the same and detectable by theobject classification module 506. In other words, the objectclassification module 506 does more than simple extension filtering.

The object classification module 506 may also determine whether eachobject should be stored or discarded. This determination is based ondefinable capture rules used by the object classification module 506.For example, a capture rule may indicate that all Web traffic is to bediscarded. Another capture rule could indicate that all PowerPointdocuments should be stored except for ones originating from the CEO's IPaddress. Such capture rules may be implemented as regular expressions orby other similar means.

The capture rules may be authored by users of a capture system. Thecapture system may also be made accessible to any network-connectedmachine through the network interface module 500 and/or user interface510. In one embodiment, the user interface 510 is a graphical userinterface providing the user with friendly access to the variousfeatures of the capture system 512. For example, the user interface 510may provide a capture rule authoring tool that allows any capture ruledesired to be written. These rules are then applied by the objectclassification module 506 when determining whether an object should bestored. The user interface 510 may also provide pre-configured capturerules that the user selects from along with an explanation of theoperation of such standard included capture rules. Generally, bydefault, the capture rule(s) implemented by the object classificationmodule 506 captures all objects leaving the network that the capturesystem is associated with.

If the capture of an object is mandated by one or more capture rules,the object classification module 506 may determine where in the objectstore module 508 the captured object should be stored. FIG. 7illustrates an embodiment of an object store module. Within the contentstore 702 are files 704 grouped up by content type. Thus, for example,if an object classification module (such as object classification module506) determines that an object is a Word document that should be stored,it can store it in the file 704 reserved for Word documents. The objectstore module 706 may be internal to a capture system or external(entirely or in part) using, for example, some network storage techniquesuch as network attached storage (NAS), and storage area network (SAN),or other database.

In an embodiment, the content store 702 is a canonical storage locationthat is simply a place to deposit the captured objects. The indexing ofthe objects stored in the content store 702 is accomplished using a tagdatabase 700. The tag database 700 is a database data structure in whicheach record is a “tag” that indexes an object in the content store 702and contains relevant information (metadata) about the stored object. Anexample of a tag record in the tag database 700 that indexes an objectstored in the content store 702 is set forth in Table 1:

TABLE 1 Field Name Definition (Relevant Information) MAC Address NIC MACaddress Source IP Source IP Address of object Destination IP DestinationIP Address of object Source Port Source port number of objectDestination Port Destination port number of the object Protocol Protocolthat carried the object Instance Canonical count identifying objectwithin a protocol capable of carrying multiple data within a singleTCP/IP connection Content Content type of the object Encoding Encodingused by the protocol carrying object Size Size of object Timestamp Timethat the object was captured Owner User requesting the capture of object(possibly rule author) Configuration Capture rule directing the captureof object Signature Hash signature of object Tag Signature Hashsignature of all preceding tag fields

There are various other possible tag fields and some tag fields listedin Table 1 may not be used. In an embodiment, the tag database 500 isnot implemented as a database and another data structure is used.

The tag fields shown in Table 1 can be expressed more generally, toemphasize the underlying information indicated by the tag fields invarious embodiments. Some of these possible generic tag fields are setforth in Table 2:

TABLE 2 Field Name Definition Device Identity Identifier of capturedevice Source Address Origination Address of object Destination AddressDestination Address of object Source Port Origination Port of objectDestination Port Destination Port of the object Protocol Protocol thatcarried the object Instance Canonical count identifying object within aprotocol capable of carrying multiple data within a single connectionContent Content type of the object Encoding Encoding used by theprotocol carrying object Size Size of object Timestamp Time that theobject was captured Owner User requesting the capture of object (ruleauthor) Configuration Capture rule directing the capture of objectSignature Signature of object Tag Signature Signature of all precedingtag fields

For many of the above tag fields in Tables 1 and 2, the definitionadequately describes the relational data contained by each field. Forthe content field, the types of content that the object can be labeledas are numerous. Some example choices for content types (as determined,in one embodiment, by the object classification module 30) are JPEG,GIF, BMP, TIFF, PNG (for objects containing images in these variousformats); Skintone (for objects containing images exposing human skin);PDF, MSWord, Excel, PowerPoint, MSOffice (for objects in these popularapplication formats); HTML, WebMail, SMTP, FTP (for objects captured inthese transmission formats); Telnet, Rlogin, Chat (for communicationconducted using these methods); GZIP, ZIP, TAR (for archives orcollections of other objects); Basic_Source, C++_Source, C_Source,Java_Source, FORTRAN_Source, Verilog_Source, VHDL_Source,Assembly_Source, Pascal_Source, Cobol_Source, Ada_Source, Lisp_Source,Perl_Source, XQuery_Source, Hypertext Markup Language, Cascaded StyleSheets, JavaScript, DXF, Spice, Gerber, Mathematica, Matlab, AllegroPCB,ViewLogic, TangoPCAD, BSDL, C_Shell, K_Shell, Bash_Shell, Bourne_Shell,FTP, Telnet, MSExchange, POP3, RFC822, CVS, CMS, SQL, RTSP, MIME, PDF,PS (for source, markup, query, descriptive, and design code authored inthese high-level programming languages); C Shell, K Shell, Bash Shell(for shell program scripts); Plaintext (for otherwise unclassifiedtextual objects); Crypto (for objects that have been encrypted or thatcontain cryptographic elements); Englishtext, Frenchtext, Germantext,Spanishtext, Japanesetext, Chinesetext, Koreantext, Russiantext (anyhuman language text); Binary Unknown, ASCII Unknown, and Unknown (ascatchall categories).

The mapping of tags to objects may be obtained by using uniquecombinations of tag fields to construct an object's name. For example,one such possible combination is an ordered list of the source IP,destination EP, source port, destination port, instance and timestamp.Many other such combinations including both shorter and longer names arepossible. A tag may contain a pointer to the storage location where theindexed object is stored.

The objects and tags stored in the object store module 508 may beinteractively queried by a user via the user interface 510. In oneembodiment, the user interface interacts with a web server (not shown)to provide the user with Web-based access to the capture system 312. Theobjects in the object store module 508 are searchable for specifictextual or graphical content using exact matches, patterns, keywords,and/or various other attributes.

For example, the user interface 510 may provide a query-authoring tool(not shown) to enable users to create complex searches of the objectstore module 308. These search queries are provided to a data miningengine (not shown) that parses the queries the object store module. Forexample, tag database 700 may be scanned and the associated objectretrieved from the content store 702. Objects that matched the specificsearch criteria in the user-authored query are counted and/or displayedto the user by the user interface 510.

Searches may be scheduled to occur at specific times or at regularintervals. The user interface 510 may provide access to a scheduler (notshown) that periodically executes specific queries. Reports containingthe results of these searches are made available to the user at runtimeor at a later time such as generating an alarm in the form of an e-mailmessage, page, system log, and/or other notification format.

Generally, a capture system has been described above as a stand-alonedevice. However, capture systems may be implemented on any appliancecapable of capturing and analyzing data from a network. For example, thecapture system 510 described above could be implemented on one or moreof the servers or clients shown in FIG. 1. Additionally, a capturesystem may interface with a network in any number of ways includingwirelessly.

Document Registration

The capture system described above implements a document registrationscheme. A user registers a document with a capture system, the systemthen alerts the user if all or part of the content in the registereddocument is attempting to, or leaving, the network. Thus, un-authorizeddocuments of various formats (e.g., Microsoft Word, Excel, PowerPoint,source code of any kind, text are prevented) are prevented from leavingan enterprise. There are great benefits to any enterprise that keeps itsintellectual property, and other critical, confidential, or otherwiseprivate and proprietary content from being mishandled. Sensitivedocuments are typically registered with the capture system 200, althoughregistration may be implemented using a separate device.

FIG. 8 illustrates an embodiment of a capture/registration system. Thecapture/registration system 800 has components which are used in asimilar number similar or identical to the capture system 500 shown inFIG. 5, including the network interface module 602, the object storemodule 806, user interface 812, and object capture modules 804 (thepacket capture 502, object assembly 504, and object classification 506modules of FIG. 5).

The capture/registration system 800 includes a registration module 810interacting with a signature storage 808 (such as a database) to helpfacilitate a registration scheme. There are numerous ways to registerdocuments. For example, a document may be electronically mailed(e-mailed), uploaded to the registration system 800 (for example throughthe network interface module 1002 or through removable media), theregistration system 800 scanning a file server (registration server) fordocuments to be registered, etc. The registration process may beintegrated with an enterprise's document management systems. Documentregistration may also be automated and transparent based on registrationrules, such as “register all documents,” “register all documents byspecific author or IP address,” etc.

After being received, classified, etc., a document to be registered ispassed to the registration module 810. The registration module 810calculates a signature or a set of signatures of the document. Asignature associated with a document may be calculated in various ways.An exemplary signature consists of hashes over various portions of thedocument, such as selected or all pages, paragraphs, tables andsentences. Other possible signatures include, but are not limited to,hashes over embedded content, indices, headers, footers, formattinginformation, or font utilization. A signature may also includecomputations and meta-data other than hashes, such as word RelativeFrequency Methods (RFM)—Statistical, Karp-RabinGreedy-String-Tiling-Transposition, vector space models, diagrammaticstructure analysis, etc.

The signature or set of signatures associated on a document is stored inthe signature storage 808. The signature storage 808 may be implementedas a database or other appropriate data structure as described earlier.In an embodiment, the signature storage 808 is external to the capturesystem 800.

Registered documents are stored as objects in the object store module806 according to the rules set for the system. In an embodiment, onlydocuments are stored in the content store 806 of the object systemnetwork. These documents have no associated tag since many tag fields donot apply to registered documents.

As set forth above, the object capture modules 802 extract objectsleaving the network and store various objects based on capture rules. Inan embodiment, all extracted objects (whether subject to a capture ruleor not) are also passed to the registration module for a determinationwhether each object is, or includes part of, a registered document.

The registration module 810 calculates the set of one or more signaturesof an object received from the object capture modules 804 in the samemanner as the calculation of the set of one or more signatures of adocument received from the user interface 812 to be registered. This setof signatures is then compared against all signatures in the signaturedatabase 808. However, parts of the signature database may be excludedfrom a search to decrease the amount comparisons to be performed.

A possible unauthorized transmission is detectable if any one or moresignatures in the set of signatures of an extracted object matches oneor more signatures in the signature database 808 associated with aregistered document. Detection tolerances are usually configurable. Forexample, the system may be configured so that at least two signaturesmust match before a document is deemed unauthorized. Additionally,special rules may be implemented that make a transmission authorized(for example, if the source address is authorized to transmit anydocuments off the network).

A query generator 614 may be used to search the object store module 806for specific documents, emails, etc.

An embodiment of a registration module is illustrated in FIG. 9. Asdiscussed above, a user may select a document to be registered. Theregistration engine 902 generates signatures for the document andforwards the document to content storage and the generated signatures tothe signature database 808. Generated signatures are associated with adocument, for example, by including a pointer to the document or to someattribute to identify the document.

The registration engine calculates signatures for a captured object andforwards them to the search engine 910. The search engine 910 queriesthe signature database 808 to compare the signatures of a capturedobject to the document signatures stored in the signature database 808.Assuming for the purposes of illustration, that the captured object is aWord document that contains a pasted paragraph from registeredPowerPoint document, at least one signature of registered PowerPointsignatures will match a signature of the captured Word document. Thistype of event is referred to as the detection of an unauthorizedtransfer, a registered content transfer, or other similarly descriptiveterm.

When a registered content transfer is detected, the transmission may behalted or allowed with or without warning to the sender. In the event ofa detected registered content transfer, the search engine 910 mayactivate the notification module 912, which sends an alert to theregistered document owner. The notification module 912 may senddifferent alerts (including different user options) based on the userpreference associated with the registration and the capabilities of theregistration system.

An alert indicates that an attempt (successful or unsuccessful) totransfer a registered content off the network has been made.Additionally, an alert may provide information regarding the transfer,such as source IP, destination IP, any other information contained inthe tag of the captured object, or some other derived information, suchas the name of the person who transferred the document off the network.Alerts are provided to one or more users via e-mail, instant message(IM), page, etc. based on the registration parameters. For example, ifthe registration parameters dictate that an alert is only to be sent tothe entity or user who requested registration of a document then noother entity or user will receive an alert.

If the delivery of a captured object is halted (the transfer is notcompleted), the user who registered the document may need to provideconsent to allow the transfer to complete. Accordingly, an alert maycontain some or all of the information described above and additionallycontain a selection mechanism, such as one or two buttons—to allow theuser to indicate whether the transfer of the captured object is eligiblefor completing. If the user elects to allow the transfer, (for example,because he is aware that someone is emailing a part of a registereddocument (such as a boss asking his secretary to send an email), thetransfer is executed and the captured object is allowed to leave thenetwork.

If the user disallows the transfer, the captured object is not allowedoff of the network and delivery is permanently halted. Several haltingtechniques may be used such as having the registration system proxy theconnection between the network and the outside, using a black holetechnique (discarding the packets without notice if the transfer isdisallowed), a poison technique (inserting additional packets onto thenetwork to cause the sender's connection to fail), etc.

Attributes

When a search of object captured by a capture system is performed, it isdesirable to make the search as fast as possible. A technique to speedup searches is to perform searches over the tag database instead of thecontent store, since the content store will generally be stored on diskand is therefore far more costly both in terms of time and processingpower.

A query is generally in the form of a regular expression. A regularexpression is a string that describes or matches a set of stringaccording to certain syntax rules. There are various well-known syntaxrules such as POSIX standard regular expressions and PERL scriptinglanguage regular expressions and are used by many text editors andutilities to search and manipulate bodies of text based on certainpatterns and are well-known in the art. For example, according to onesyntax (Unix), the regular expression 4\d{15} means the digit “4”followed by any fifteen digits in a row. This user query would returnall objects containing that match this pattern.

Certain useful search categories are not well defined by a singleregular expression. For example, a query of all emails containing acredit card number is hard to define with a single regular expression asvarious. Credit card companies use different numbering patterns andconventions. However, a card number for each company is representable bythe regular expression and the generic concept of “credit card number”is representable by the union of all such regular expressions.

An attribute, represents a group of one or more regular expressions (orother such patterns). The term “attribute” is merely descriptive, andcould just as easily be termed “category,” “regular expression list,” orany other descriptive term.

Attribute tagging is performed in the object classification 506 moduledescribed above. However, attribute tagging may be implemented in otherparts of the capture system 512 or as a separate module.

An embodiment of the object classification module is illustrated in FIG.10. Objects arriving from the object assembly module 504 are forwardedto the content store 702 and are used to generate tags to be associatedwith the objects. The content classifier 1002 determines the contenttype of the object. The content type is then forwarded to the taggenerator 1008 where it is inserted into the content field describedabove. Various other tasks, such as protocol and size determination, arerepresented by the other processing block 1006.

The attribute module 1004 generates an attribute index that isinsertable into an index field of the tag by the tag generator 1008.

FIG. 11 illustrates an exemplary attribute index. A plurality of regularexpressions (labeled RegEx 1100-1104) are mapped to attributes using theattribute map 1106. For example, if regular expressions RegEx 1100-1102can represent credit card patterns, then these regular expressions wouldmap to a credit card number attribute. Regular expressions 1103 and 1104may represent phone number patterns and would map to a phone numberattribute. A mapping of a regular expression to an attribute is thus thereservation and usage of that attribute as implying a successfulmatching of the regular expression.

Attribute index 1106 is used to represent the attributes in a compactform. The attribute index 1108 may be implemented as a bit vector with avector of bits having one bit position associated with each definedattribute. In one embodiment, the attribute index 1108 is 128 bits and128 separate attributes are definable with this index and occurindependently of one another.

The association of attributes to bit positions may be maintained in atable. For example, such a table may associate bit position A with thecredit card number attribute and bit position B with the phone numberattribute. Since, in this example, regular expressions 1100-1102 map tothe credit card attribute, observing any one of the patterns defined byRegEx 1100-1102 causes an a captured object bit position A to be set toshow the presence of a credit card number in the captured object.

Setting a bit position is done by changing a bit either from “0” to “1”or from “1” to “0” depending on which value is the default. In oneembodiment, bit positions are initialized as “0” and are set to “1” toshow the presence of an attribute. Similarly, since regular expressions1103 and 1104 map to the phone number attribute, observing any one ofthe patterns defined by RegEx 1103 or 1104 causes bit position B to beset to show the presence of a phone number in the captured object.

An embodiment of the attribute module is illustrated in FIG. 12. Theinput of the attribute module 1004, as set forth above, is an objectcaptured by the object capture and assembly modules. The object may be aword document, email, spreadsheet, or some other document that includestext or other characters that represent a pattern expressed as a regularexpression.

The text content contained in the object may be extracted to simplifythe attribute tagging processing. The text content of objects includesonly textual characters without formatting or application context. Theobject or text extracted from an object is provided to parser 1200. Theparser 1200 parses the object to identify which regular expressionsappear in the object.

The parser 1200 accesses a regular expression table 1202 that lists allthe regular expressions of interest. The parser 1200 then determineswhich of the regular expressions appear in the object or the textextracted from the object.

The regular expression table 1202 also associates each regularexpression contained therein with an attribute. In this manner, theregular expression table 1202 can function as the regular expression toattribute map 1106 of FIG. 11. For example, the regular expression table1202 as shown in FIG. 12 maps regular expression A to attribute X;regular expressions B and C to attribute Y; and regular expressions D,E, and F to attribute Z.

Since the regular expression table 1202 contain the regular expressionsand their attribute mapping, the parser 1200, by parsing the regularexpressions over the object determines which attributes are present inan object. In one embodiment, the parsing is done faster by parsing onlythe regular expressions related to attributes that have not yet beenfound in the object. For example, if the parser finds a hit from regularexpression D in the object, then attribute Z is found in the object.This makes parsing using regular expressions E and F unnecessary, sinceattribute Z is already hit.

The parser 1200 outputs a list of attributes found in an object. Asexplained above, an attribute is a category of patterns such as creditcard number, phone numbers, email addresses, bank routing numbers,social security numbers, confidentiality markers, web sites, the namesof executive officers of a company, medical conditions or diagnoses,confidential project names or numerical strings indicating salary orcompensation information.

Attributes found in the object are provided to an index generator 1204.The index generator 1204 generates the attribute index 1108 describedwith reference to FIG. 11. The index generator 1204 accesses anattribute table 1206 which contains a mapping of attributes to bitpositions of the attribute index 1108. For example, in FIG. 12,attribute X is mapped to bit position 1, attribute Y is mapped to bitposition 2, and attribute Z is mapped to bit position 3.

If an object contained regular expression A, D, and F, then the parser1200 would first note that attribute X has been hit. When recognizingregular expression D, the parser 1200 would note that attribute Z hasbeen hit. Since these are the only attributes in this abbreviatedexample, the parser 1200 would provide attributes X and Z to the indexgenerator 1204. According to the attribute table 1206, the indexgenerator would set bit positions 1 and 3 of an attribute index 1108.Thus, for this simplified example, the attribute index 1108 would be“101” first bit positions 1 through 3.

The generation of an attribute index 1108 and the use of the specificmapping tables shown in FIG. 12 is just one example of an attributemodule 1004 performing attribute tagging. In another embodiment, eachpossible attribute has a separate field in the tag associated with theobject indicating whether the attribute is present in the object. Thus,an attribute index may be thought of as a summary of a plurality ofattribute fields. Alternatively, each bit position of the attributeindex may be thought of as a separate field. Various otherimplementations and visualizations are also possible.

An embodiment of a method for attribute tagging is described by FIG. 13.In block 1302, an object is captured. In block 1304, the textual contentis extracted from the object. In block 1306, a determination is made asto whether a regular expression appears in the extracted text.

If the regular expression underconsideration does not appear in thetext, then, processing continues again at block 1306 using the nextregular expression on the regular expression list. If, however, theregular expression under consideration does appear in the text, then, inblock 1308 the attribute associated with the regular expression istagged. This may be done by setting a field or position in an index in atag of metadata associated with the object.

In block 1310, all other regular expressions associated with theobserved attribute are removed from future consideration with respect tothe object. In block 1312, a determination is made as to whetherattribute tagging has completed with respect to the object. If noregular expressions remain to be compared with the extracted text, thenthe attribute tagging is complete and processing terminates. Otherwise,processing continues at block 1306 with the next regular expression onthe list evaluated.

FIG. 14 illustrates an exemplary flow of querying captured objects. Inblock 1402, a query is issued. The query may be received by a capturedevice via a user interface. The process described with reference toFIG. 13 may be implemented entirely within the user interface, withinsome query module of the user interface, or a separate query module.

The query—in addition to other limitations, such as content type, size,time range, and so on—may contain one or more attributes the query islooking for. For example, the query could be for all Microsoft Exceldocuments from last week containing credit card numbers (credit cardnumbers being an attribute).

The received query may only include one or more regular expressions, asshown in block 1404. In block 1406, the regular expression is matched toan attribute, if possible. For example, if the regular expression in thequery is only satisfied if another regular expression associated with anattribute is satisfied, then, objects having this attribute tagged aremore relevant for this query than objects in general. In particular, anyobject satisfying the regular expression would also satisfy theattribute. For example, a query for a specific credit card number orrange will satisfy the credit card attribute.

Whether provided by the user, or identified based on the query, in block1408, the appropriate attribute or attributes are used to eliminateobjects from the query. In one embodiment, a search is done over theappropriate attribute field or index bit positions in the tags in thetag database. If the attributes being sought are not shown as present inan object, the object is eliminated from further consideration for thisquery.

In block 1410, the objects remaining after elimination at 1408 areretrieved from the medium they are stored on (such as a disk) intomemory. They can now be presented to the user as query results, orobject can be further eliminated by parsing the retrieved objects forthe specific regular expression queried for, where no specific attributewas named. Alternatively, only a link to the objects remaining afterelimination are retrieved.

In one embodiment, the attributes are completely user-configurable. Auser interface provides an attribute editor that allows a user to defineattributes by creating an attribute and associating a group of one ormore regular expressions with the created attribute. The capture devicemay come pre-configured with a list of common or popular attributes thatmay be tailored specifically to the industry into which the capturedevice is sold.

In one embodiment, a capture device may create new attributesautomatically. For example, a capture device may observe that a certainregular expression is being searched with some threshold frequency(generally set to be above normal). The capture device creates anattribute to be associated with this regular expression and beginstagging the newly defined attribute when capturing new objects. Inanother embodiment, a capture device may suggest that a new attribute becreated when a regular expression is searched frequently. In yet anotherembodiment, a capture device may suggest that an attribute be deleted ifinfrequently used to make room for another more useful attribute.

Query Generation

Objects and/or their associated metadata should be searchable uponrequest. For example, emails, documents, images, etc. may be processedby a capture system and searched. FIG. 15 illustrates an exemplaryembodiment of a capture system for querying captured data. The networkinterface module 500, packet capture module 502, object assembly module504, object classification module 506, object store module 508, and userinterface 510 have all been described before.

The capture system 1512 also includes a query generator module 1514. Thequery generator module 1514 changes a search string into a capturesystem 1512 usable form. A search string comes from the user interface1510 or an outside source, such as another capture system or remoterequest. The user interface 1510 may be outside of the capture system1512. The query generator module 1514 may also be used with acapture/registration system (such as the one illustrated in FIG. 8).

As described earlier, objects and tags are generally stored in theobject store module 508 even though objects and/or tags may also bepersisted to remote databases. These objects and tags are searchable forspecific content using exact matches, patterns, keywords, and/or variousother attributes generated by the query generator 1514. For example, thetag database 700 may be scanned and the associated object retrieved fromthe content store 702. Additionally, the query generator 1514 may alsosearch “look-asides” such as dictionaries and compiled lists.Look-asides may be stored in volatile storage (such as RAM) and/or innon-volatile storage (such as a hard disk, flash device, etc.). Inpractical deployments, a volatile look-aside structure is often shadowedinto a non-volatile storage for persistence over a power loss or otherfault condition.

Searches may be conducted at specific times or be periodicallyscheduled. For example, in some business environments it may bebeneficial to have an on-demand query about emails leaving the networkfrom a key employee and have a weekly search and report.

The user interface 510 may provide access to a scheduler (not shown)that periodically executes specific queries. Reports containing theresults of these searches are made available to the user at runtime orat a later time by generating an alarm in the form of an e-mail message,page, system log, and/or other notification format.

Capture (and capture/registration) systems receive, sort, and store manydifferent types of data. In one embodiment, captured objects, theirmetadata, and/or look-aside information are stored based on the timereceived. Accordingly, captured objects, metadata, and/or look-asideinformation is already in an order. Because information is alreadysorted by time, a search of this information is an order n search whichis an improvement over the prior art in which time is not a primaryindex. Storing captured objects by time may also be accomplished bypartitioning into time divisions. Of course, more than one entry may bemade per time division. Time divisions may be of any size. For example,the time division may be one hour, two hours, one day, etc.

FIGS. 16( a)-(c) illustrate an embodiment of an exemplary storageconfiguration for data received by a capture system. In this particularexample, time divisions (T_(N)) are used.

FIG. 16( a) illustrates the storage configuration prior to any eventsoccurring. Metadata (tag) storage 1601 stores the metadata associatedwith a captured object. Exemplary metadata has been describedpreviously. Metadata may be separated by categories. Metadata storage1601 is divided into five categories: email, documents, protocols,images, and other/miscellaneous. However, any number of classificationsmay be used. A single transaction could fall into more than oneclassification. For example, an email with an attached word documentfalls into three categories: email, document, and protocol (the protocolis dependent upon the type of email service used such as SMTP, POP3 orIMAP). Typically, a disk (such as a hard drive) or several disks forredundancy and performance reasons, are used for metadata storage. Tagdatabase 700 is an example of metadata storage.

Look-aside storage 1603 stores information/data known to a capturesystem from a source other than object capture. For example, look-asidestorage 1603 may include such information as the name of the competitorto company A is company B, the names of people in a particulardepartment or office, etc. Volatile storage (such as RAM) ornon-volatile storage (such as a disk) may be used for look-aside storage1603. If volatile storage is used, provisions should be made to backupthis information in the look-aside storage 1603 into non-volatilestorage in case of a failure (such as a power failure) that would causethe information in the volatile storage to be lost. If time divisionsare used, time divisions of look-aside storage 1603 could have differentvalues.

Object storage 1605 stores captured/reassembled objects. For example,object storage 1605 may store text documents, images, videos, etc. Thecontent store 702 is an example of object storage.

FIG. 16( b) shows the occurrence of a single event during T₁. An emailwith an attached document is sent to a competitor in the look-asidedictionary during this time division. This email is processed by thecapture system and, accordingly, all three storages have an entry forthat email. In the metadata data storage 1601, an entry for the emailand an entry for the attached document are created. The look-asidestorage 1603 has an entry noting that a particular competitor in thelook-aside dictionary was sent the email. Finally, the email text andthe attachment are saved in the object storage 1605.

FIG. 16( c) shows the occurrence of a two events during T₃. The firstevent is uploading an image to a web page using the HTTP protocol. As aresult, there are entries in the image and protocol (HTTP) categories ofthe metadata storage 1601 and in the object storage 1605 during T₃. Thesecond event of T₃ is an email being received with no attachments.Accordingly, an entry exists in the email category of the metadatastorage 1601 and the email text is stored in the object storage 1605.

FIG. 17 illustrates an embodiment of a method for performing a query ina capture system. At 1701, a query is received or generated. A receivedquery normally comes into a capture system via a user interface. Thesequeries are typically written in a form that is not standard to thecapture system and must be further decomposed. An exemplary query is“Find a MS Office document over 50 KB in size with the keywords“Confidential” and “Project” in the document with a destination addressof “Competitor1” or “Competitor2” that was sent between January 1 andJanuary 3.”

This query is decomposed into search terms and types at 1703. For eachof these search terms, a list of matches to stored data will begenerated. For the above example query, the decomposition results in sixlists that will be generated: 1) attachments by time, content-type, andsize; 2) keyword “Confidential;” 3) keyword “Project;” 4) email by time,content-type, and size; 5) keyword “Competitor1;” and 6) keyword“Competitor2.” Of course, the exact decomposition may very. For example,keywords could be combined, etc. Search terms may also be tokenizeddepending upon where the search term is located. Tokens are used in thesearching of the tag database as described earlier. The exacttokenization method (how strings of text are broken into individualtokens) is not important as long as that method is identical between thecapture and searching portions of the device.

The appropriate location(s)/file(s) to search are determined at 1705. Asdescribed earlier, typical capture systems use different types ofstorage locations and types. The search terms decomposed at 1703 mayrelate to these storage locations and types (such as email orattachment) differently based on the context of the search query. Forexample, a search for a keyword may typically start with a search in alook-aside such as a dictionary. Since these searches (searches oflook-asides) are generally fast, a quick determination of whether or notthe capture system has ever searched for these keywords and if acorresponding entry in metadata would have been made. Whereas theappropriate search for attachments in general is the metadata storage orobject storage. Typically, the metadata storage is searched prior toobject storage because it would be faster.

The appropriate search location(s)/file(s) are searched at 1707. Forexample, a look-aside may be searched for a keyword and the metadatastorage searched for an attachment over 50 KB in size. Someoptimizations for searching may be utilized by a capture system. Oneoptimization is to perform a “dirty search” such as only searching for aportion of a keyword. If the search has several requirements, and onefails, it may be beneficial to not have searched for the entire keyword.However, further refinements of the search may need to be performed at alater time (for example, if no other search requirements fail then theentire keyword will have to be searched for).

Another optimization is that if one term of an “OR” clause of a searchhas been found, then the other terms of the “OR” clause do not need tobe searched for. For example, of the search is for “A or B” and “A” isfound, then “B” does not need to be searched for because the clause hasalready been satisfied.

Another optimization is to first search the list of terms that could bein the metadata. If a particular value is not going to be in themetadata then the metadata should not be searched. In fact, if a valueis not in the metadata the entire object storage may have to be searchedfor that value. This would be a relatively slower search. For example, adetermination of whether or not a capture system was previouslysearching and creating metadata for the text “Butters” is generallyfaster than searching the metadata storage.

In an embodiment, searches are performed one time division at a time.For example, if the capture system has 24 time divisions (one divisionper hour of the day), each hour is searched individually. Search resultsfrom the same time division are more likely to be related than searchresults from different time divisions. For example, an email that meetsthe criteria of the exemplary search query is likely to be processed andhave items (metadata and objects) stored in the same time division.Accordingly, search results from different time divisions are often notcombinable. Searching by time division may result in a search time offaster than order n.

A list of matches for each search is generated at 1709. Lists of matchesare subsets of what is stored in a particular storage location. Forexample, a list of matches from the metadata storage is those metadatathat match the search criteria. If time divisions are used, then eachlist of matches is divided by time division.

The lists of matches for each search are intersected at 1711.Intersecting determines the result of the search. If time divisions areused, results from each list for a specific time division areintersected with results from the same time division.

Searching, matching, and intersecting may also be staggered so as tobetter utilize resources. For example, look-asides may be searchedbefore metadata or object storage. If there is no match in a look-aside,it may not be useful to search the other databases as those searcheswould likely result in resource waste. Likewise, object storage may besearched prior to object storage.

This staggering may also apply to searching, matching, and intersectingby time division in a “pipeline” fashion. FIG. 18 illustrates pipelinestaggering. At time T₁ 1801, a first time slot 1805 is searched. At timeT₂ 1803, a second time slot 1807 may be searched while a list (or lists)of matches is being generated for the first time slot, etc. The twostaggering optimizations may also be combined.

FIG. 19 illustrates the exemplary query being performed. This query maybe made over a particular time division or over the entire time that thecapture system was operating.

The query is broken into different keywords and search types at 1901. Asshown, attachments and emails (by time, content, and size) and keywordswill be searched for.

The appropriate storage locations have been determined at 1903. Themetadata storage will be searched for attachments and emails, while thelook-aside storage will be searched for keywords.

Lists have been generated based on a search of these locations at 1905.Lists A and D have been generated from the search of the metadatastorage. Lists B, C, F, and G have been generated from searching thelook-aside storage. As described above, the searching and generation oflists may be staggered. For example, the look-aside storage may besearched before searching the metadata storage. These searches and listgenerations may be performed by time division. The object storage mayalso have to be searched. For example, the object storage may have to besearched to retrieve an email to look for the destination address of“Competitor 1” or “Competitor 2.”

Lists are intersected at 1907. Lists B and C are interested with A; Fand G are intersected with D; and the results of those two previousintersections are intersected to create the final search result. Again,the timing of these intersections may be staggered.

Closing Comments

An article of manufacture may be used to store program code. An articleof manufacture that stores program code may be embodied as, but is notlimited to, one or more memories (e.g., one or more flash memories,random access memories (static, dynamic or other)), optical disks,CD-ROMs, DVD ROMs, EPROMs, EEPROMs, magnetic or optical cards or othertype of machine-readable media suitable for storing electronicinstructions. Program code may also be downloaded from a remote computer(e.g., a server) to a requesting computer (e.g., a client) by way ofdata signals embodied in a propagation medium (e.g., via a communicationlink (e.g., a network connection)).

In one embodiment, a capture system is an appliance constructed usingcommonly available computing equipment and storage systems capable ofsupporting the software requirements.

FIG. 20 shows an embodiment of a computing system (e.g., a computer).The exemplary computing system of FIG. 20 includes: 1) one or moreprocessors 2001; 2) a memory control hub (MCH) 2002; 3) a system memory2003 (of which different types exist such as DDR RAM, EDO RAM, etc,); 4)a cache 2004; 5) an I/O control hub (ICH) 2005; 6) a graphics processor2006; 7) a display/screen 2007 (of which different types exist such asCathode Ray Tube (CRT), Thin Film Transistor (TFT), Liquid CrystalDisplay (LCD), Digital Light Processing (DLP), Organic LED (OLED), etc.;and 8) one or more I/O and storage devices 2008.

The one or more processors 2001 execute instructions in order to performwhatever software routines the computing system implements. Theinstructions frequently involve some sort of operation performed upondata. Both data and instructions are stored in system memory 2003 andcache 2004. Cache 2004 is typically designed to have shorter latencytimes than system memory 2003. For example, cache 1304 might beintegrated onto the same silicon chip(s) as the processor(s) and/orconstructed with faster SRAM cells whilst system memory 2003 might beconstructed with slower DRAM cells. By tending to store more frequentlyused instructions and data in the cache 2004 as opposed to the systemmemory 2003, the overall performance efficiency of the computing systemimproves.

System memory 2003 is deliberately made available to other componentswithin the computing system. For example, the data received from variousinterfaces to the computing system (e.g., keyboard and mouse, printerport, LAN port, modem port, etc.) or retrieved from an internal storageelement of the computing system (e.g., hard disk drive) are oftentemporarily queued into system memory 2003 prior to their being operatedupon by the one or more processor(s) 2001 in the implementation of asoftware program. Similarly, data that a software program determinesshould be sent from the computing system to an outside entity throughone of the computing system interfaces, or stored into an internalstorage element, is often temporarily queued in system memory 1303 priorto its being transmitted or stored.

The ICH 2005 is responsible for ensuring that such data is properlypassed between the system memory 2003 and its appropriate correspondingcomputing system interface (and internal storage device if the computingsystem is so designed). The MCH 2002 is responsible for managing thevarious contending requests for system memory 2003 access amongst theprocessor(s) 2001, interfaces and internal storage elements that mayproximately arise in time with respect to one another.

One or more I/O devices 2008 are also implemented in a typical computingsystem. I/O devices generally are responsible for transferring data toand/or from the computing system (e.g., a networking adapter); or, forlarge scale non-volatile storage within the computing system (e.g., harddisk drive). ICH 2005 has bi-directional point-to-point links betweenitself and the observed I/O devices 2008. A capture program,classification program, a database, a filestore, an analysis engineand/or a graphical user interface may be stored in a storage device ordevices 2008 or in memory 2003.

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. It will, however,be evident that various modifications and changes may be made theretowithout departing from the broader spirit and scope of the invention asset forth in the appended claims. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense.

Thus, a capture system and a document/content registration system havebeen described. In the forgoing description, various specific valueswere given names, such as “objects,” and various specific modules, suchas the “registration module” and “signature database” have beendescribed. However, these names are merely to describe and illustratevarious aspects of the present invention, and in no way limit the scopeof the present invention. Furthermore, various modules, may beimplemented as software or hardware modules, combined or withoutdividing their functionalities into modules at all. The presentinvention is not limited to any modular architecture either in softwareor in hardware, whether described above or not.

1. A method, comprising: receiving a query at a system, which includes aprocessor and a memory; searching for a plurality of search terms of thequery; generating lists of matches for the search terms; intersectingthe lists of matches; providing a response to the query based, at least,on the intersecting of the lists; storing network transmitted objectsaccording to a capture rule that defines which objects are to becaptured by the system; and indexing the objects such that they can beevaluated in response to receiving a subsequent query.
 2. The method ofclaim 1, wherein a plurality of locations is searched, and wherein thelocations are part of a group of locations, the group consisting of: a)a look-aside storage; b) a metadata storage; and c) an object storage.3. The method of claim 2, wherein the look-aside storage, the metadatastorage, and the object storage are presorted by time.
 4. The method ofclaim 2, wherein the look-aside storage, the metadata storage, and theobject storage are partitioned into time divisions.
 5. The method ofclaim 1, further comprising: intercepting data from data streams; andreconstructing the data.
 6. The method of claim 1, wherein the searchingis performed in a staggered pipeline manner such that a first time slotassociated with one of the search terms is searched while a list for adifferent one of the search terms is generated for a second time slot.7. The method of claim 1, wherein the searching is executed by searchingthrough a metadata storage that includes e-mails, and by searchingthrough a look-aside storage for the search terms.
 8. Logic encoded innon-transitory media for performing operations, comprising: receiving aquery at a system; searching for a plurality of search terms of thequery; generating lists of matches for the search terms; intersectingthe lists of matches; providing a response to the query based, at least,on the intersecting of the lists; storing network transmitted objectsaccording to a capture rule that defines which objects are to becaptured by the system; and indexing the objects such that they can beevaluated in response to receiving a subsequent query.
 9. The logic ofclaim 8, wherein a plurality of locations is searched, and wherein thelocations are part of a group of locations, the group consisting of: a)a look-aside storage; b) a metadata storage; and c) an object storage.10. The logic of claim 9, wherein the look-aside storage, the metadatastorage, and the object storage are presorted by time.
 11. The logic ofclaim 9, wherein the look-aside storage, the metadata storage, and theobject storage are partitioned into time divisions.
 12. The logic ofclaim 8, the operations further comprising: intercepting data from datastreams; and reconstructing the data.
 13. The logic of claim 8, whereinthe searching is performed in a staggered pipeline manner such that afirst time slot associated with one of the search terms is searchedwhile a list for a different one of the search terms is generated for asecond time slot.
 14. The logic of claim 8, wherein the searching isexecuted by searching through a metadata storage that includes e-mails,and by searching through a look-aside storage for the search terms. 15.A system, comprising: a memory element; and a processor coupled to thememory element, wherein the system is configured for: receiving a query;searching for a plurality of search terms of the query; generating listsof matches for the search terms; intersecting the lists of matches;providing a response to the query based, at least, on the intersectingof the lists; storing network transmitted objects according to a capturerule that defines which objects are to be captured by the system; andindexing the objects such that they can be evaluated in response toreceiving a subsequent query.
 16. The system of claim 15, wherein aplurality of locations is searched, and wherein the locations are partof a group of locations, the group consisting of: a) a look-asidestorage; b) a metadata storage; and c) an object storage.
 17. The systemof claim 16, wherein the look-aside storage, the metadata storage, andthe object storage are partitioned into time divisions.
 18. The systemof claim 15, wherein the searching is performed in a staggered pipelinemanner such that a first time slot associated with one of the searchterms is searched while a list for a different one of the search termsis generated for a second time slot.
 19. The system of claim 15, whereinthe system is further configured for: intercepting data from datastreams; and reconstructing the data.
 20. The system of claim 15,wherein the searching is executed by searching through a metadatastorage that includes e-mails, and by searching through a look-asidestorage for the search terms.